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The macroscopic dynamics in separable neural networksB 

Yong Chen, Ying Hai Wang and Kong Qing Yang 
Department of Physics, Lanzhou University, Lanzhou, Gansu, 730000, China 

The parallel dynamics is given in the case of neural networks with separable coupling through 
starting from Coolen-Sherrington (CS) theory. It is shown that this retrieve dynamics as is the case 
of sequential evolution in the postulate of away from saturation and finite temperature. The finite- 
size effects is governed by a homogeneous Markov process, which differs from the time-dependent 
Ornstein-Uhlenbeck process in sequential dynamics. 
PACS number(s): 87.10. +e, 75.10.Nr, 02.50.+S 



Ising spin models for neural networks have made a more and more significant contribution to understand the 
information processing in biotic nervous system since the pioneering work by Little ffl, Hopfield and Amit et al 
H . Moreover, both far from and near saturation, starting from the theory of equilibrium statistical mechanics of 
spin-glass-like systems, a survey of the properties of the Hopfiled model is given by Amit, Gutfreund and Sompolinsky 
H Q, in the case of symmetric connections. It means that a static analysis is enough to investigate the networks with 
symmetric coupling. However, when one deals with networks with asymmetric connections, which are ubiquitous in 
real neurons of living systems || , it becomes more important to consider the dynamics rather than the equilibrium 
properties of networks. Recently, by Markovian sequential dynamics, Coolen et al. developed a series of analytic 
. scheme for networks with separable coupling which include symmetric and asymmetric Q |?J || . It is clear that there 
exist two types of deterministic dynamics, sequential and parallel, in evolution of systems. One may question: (1) 
How about the macroscopic description of this systems with synchronous case? There are difference between both 
case? (2) Additionally, how are the finite-size effect in parallel case comparing with it in sequential case presented by 
I \ Castellanos, Coolen and Viana ||? 

Since studied systems cover up the case of asymmetric conjunctions, we resort to stochastic analysis. We take up 
the Markovian dynamics, which is considered to have advantage that it not only provides a simple description of 
stochastic process but also enable to deal with nonequilibrium properties fTc[| . So, in this work we offer the foregoing 
issues based upon the way of Markov process analysis and CS theory. 

As usual, in Ising spin model of a neural networks constructed by N neurons, the states of the i-th neu- 
ron in time t is described by Si (t) E {1,-1}. Moreover, the networks has stored p sets of patterns £f <E 
{1, —1} (/i = 1, 2, . . . ,p, i — 1,2, ... , N) which are embedded for the purpose of associative memory retrieval through 
the synaptic connections with the Hebb learning rule taken into account [fllf . We take the separable interaction ma- 
trices J to be § fh| 

J v = jf E tfve? (i) 

where the A is p x p matrix representing all kinds of separable conjunctions J in common sense. In there, the coupling 
matrix, both symmetric and asymmetric, can be represented by the form of definition (Q) in most case. Further more, 
the synchronous evolution dynamics of these systems can be defined as: 



(t + At)=sgn\yj ij s j (t)\ (2) 



where At is the length of each time step. In there, the threshold of every nerve cell is set as constant one, and the 
update function is step function sgn(ir) which its value is 1 for x > and —1 for x < 0. 

Now, according to a continuous Markov process |jl3), the probability P t+At (s) of finding the system in state 
s = (si, S2, ■ ■ ■ , sn) T at time t + At is given by: 

Pt+M (s) = £V (b' s) P t (s') (3) 
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where W (s' — > s) is the so-called transfer probability or jump probability, which means finding the system in state s 
at time t + Af given it is in state s' at t time. Because of the normalization condition 

W (a' -> s) = W ( s ' s ) + W ( s ' -»■ s ') = 1 

s s^s' 

it is easy to get the following discrete mapping 

P t+At (s) = P t (s) + W (s' s) P t (s') - YJ W (s -+ s') P t (s) (4) 

When the time step At go to zero, one obtains a continuous time version of digital master equations (||): 

= YJ «, (s' - s) P t (s') - Y> (s ^ s') P t (s) (5) 

with w (s' — > s) = Jim — — -. In there, w (s — > s) represents the density of unit time for transfer probability (it 

is equal to transition rate), which the states of system change from s' to s, in the interval between t and t + At. For 
parallel evolutionary process, reflecting upon the conclusion in biological statistics, the transition rate can be given 
by @|| 



N 1 

w (s' -> s) = JT - (1 + Sl tanh (0hi (s'))) (6) 



where hi (s') = J2j Jij s j 1S local fields of a stochastic alignment of the spins, and P = ^ denotes a measure of 
the inverse magnitude of the amount of noise affecting the neurons, acting as the role of temperature in analogy to 
thermodynamic spin systems. 

Generally, we arc interested in the problem of macroscopic features rather more than the microscopic details of 
networks. Sequentially, one introduce any set of linear order variables f2& || JhJ: 

n » = ^E#*j P t (f2)=Y> t ( s )£(n-fi( S )) 11 = 1,2,. ..,p (7) 

3 s 

where Pt (St) denotes the probability of finding ft in time t. Following up the above definition and the Eq. (||), it is 

easy to get 

^P^^,(^ s )p ( ( s ')i(fi-si( s ))-^^,( s ^ s ')p t ( s )i(fi-n(s)) 

Clearly, the above master equation can be replaced by 

= EE w ( s - O p * ( s ) (* (« - « ( s ')) - s (n - si (s))) (8) 

s s' 

Following after CS theory 0, we introduce any function $ and its average is ($ = j dSlPt (St) $ (O). Its 

time differential is 

( $ («)>t = £Y>(s-s')p t ( s ) (« ( s ')) - * (« (s))] 

s s' 

= EE-(^^)p,(s)E^E E -E 



71! 

n=l /i 1 = l/i 2 = l Mp = 1 



w-»c)r^ ^'\ o.) 

where there exist appendant condition that the sum /ij = n. Inserting the unit operator J dSIS (St — St (s)) and 
then performing partial integrations yields 
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'" ' ( ° } = - E E E ^ E E • • • E [« - « wr*"' 

s s' n=l Mi— 1 M2 — 1 Mp — 1 

<9™w (s -> s') P t (s) 5 (O - O (s)) 



(10) 



In a word, this equation is the forms with Kramers-Moyal-like expansion for the master Eq. (g) for the probability 
of the introduced order variables (pattern overlaps). Under this equation with expansion forms, one can study the 
properties of pattern overlaps, describing the macroscopic behavior, with its lowest order term. 
In fact, the Eq. (TRI) is equal to 



dP t (O) 
dt 



= -E^{EE^( s O-°( s )Ms-sOP t (s)^-^s))} 



(ii) 



Consequently, at the limit case of finite temperature and far away saturation, this becomes 



dP t (n) 

dt 



E Q£l,_ { E 



s L s' 



£)[n (s') - n (s)] w (s -»• s') 



p t (s) * (n - n (s)) 



(12) 



we denote A^= | (1 — Sj tanh (/3/ij (s))) and A^= ^ (1 + Sj tanh (f3hi (s))) , corresponding with w (s, — * — Sj) and 
u> (sj — > Si) respectively. According to the definitions of order parameters Jl and the coupling matrix J, the stochastic 
local field hi (s) is equal to 

h * ( s ) = jf E = E ^ ( ^ E ^ ] = 6 ■ An 



Then, the transition rate can be rewritten as 



A;= -(l-Sitanh(/3^-An)) 



Aj= -(1 + * tanh (06 - AH)) 



Therefore, for the terms of [• • •] in the right side of Eq. ([12]), from definition (^) and (]?]), we get 



N 



A' 



E[nW-n«M--0=-^(E^)(II(i + ff 

= "4l> 



i^i A? 



JV 



n (s) + — tanh • Af2) 



JY 



(13) 



Substituting the above relation into Eq. (fi~2) yields 



jp t (o) = - ^ | p * (°) (-« (s) + a? E ^ tanh • Af2 )) } 

On the basis of Markov process theory |L7| , obviously, the deterministic Liouville forms of Eq. ( |l4| ) is 

j t % = Ihn^ f 1 ^ # tanh(/% • AH) - ft J 
with the initial value can be set as n (0) = tto = ■h Yli & s i (0)- 



(14) 



(15) 
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Comparing the Eq. ([l5|) with the deterministic evolution equations of sequential dynamics || 0, beyond our 
expect, it is clear that both case is identical completely. Obviously, the sequential dynamics with one-spin flip is a 
special case of synchronous dynamics. The origin of this equality are the premises of continuous Markov process and 
the cut of terms with higher order in the Fokkcr-Planck-type approach for master equations. From the Lindeberg 
continuous condition of Markov process jlffl , the differentia between the probabilities of contiguous states goes to zero 
faster, as the time step goes to zero. To the extent that there only exist one or several updating spin. As a result, 
the macroscopic parallel dynamics is the same as sequential dynamics in the limit case. 

Another interrelated topic is finite-size effects in networks. In another word, the problem is how large is a small 
system. As a example, in numerical simulation, it is necessary to consider system size up to N ~ 3 x 10 for 
calculating certain properties of Hopfield model p8|j . This puzzle how to account for the finite-size of the networks 
and extract useful information about the asymptotic, N — > oo, limit form networks of only a few hundred to a few 
thousand neurons, was studied by Forrest |l9[| , Kanter and Sompolinsky po| . Furthermore, in the asynchronous case, 
the more detail analysis exploration was performed by Castellanos, Coolen and Viana basing on the CS theory ||, 
with the result that the effects is governed by a time-dependent Ornstein-Uhlenbeck process in the condition of away 
from saturation. In the following context, we will investigate this effects of systems in synchronous evolutionary 
circumstance. 

Taking advantage of the above route, it is easy to get the quadric term of the right side of Eq. (|l0| ) 
53 [« (s') - « (s)f to (s s') 



• N 

E 



&«<-&tanh (fit* (11)) 



" $ (1 - tanh 2 (Phj (»))) 

A 



N 1' N 

n ~N ? 6 tanh i(3ht m) + E $ " tanh2 {Phl 



(16) 



It is straightforward to obtain the following Fokker-Planck-type equation from substituting the Eq. (|l^) into Eq. 



dp t (n) 

dt 



8 



2 dQkdtli 



1 N 11 

Ms)-]yI>tanh(/3e r An) j 



Pt(Sl) 



1 N 11 

^^^ 2 (l-tanh 2 (/3^.Af2)) | 



(17) 



Now, the diffusion parameter is different to that in sequential dynamics . It means both dynamics are more unlike 
in fluctuation or internal noise distribution. 

Following awake of the route of Castellanos et. al. |jj . the new rescaled variable q and its probability distribution 
function is defined as: 



q(i) = VN (n (t) - SI* (t)) Pt(q)= J dflPt (SI) 6 (q - Viv(n - n*) 



(18) 



where ft* is the deterministic solution of Louville equation (|T^), and Eq. (|l8|) means that the order vector ft can be 
resolved into the sum of a deterministic terms SI* and a fluctuating term with the latter terms vanishing in the limit 
of N — > oo. Moreover, from the central limit theorem, the fluctuating term can be scaled as N~i Q & 
In the limit of N — * oo, with the help (0), the Fokker-Planck-type equation of rescaled variables is deduced:. 



dPt (q) 
dt 



= -E^{ p < fo) ^ (q. *)} + 5Ei ^ to D * *)> 



2 ^ 3*9© 



(19) 



In there, the drift factor is given by 
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F (q, t) = (3 K • Aq) [1 - tanh 2 (ft ■ AO*)] ) fi - q 

+ ^lirn^ VN 1 1 tanh (/3& ■ AO*) - O* j 



(20) 



and we denote (5 (£))^ = limjv^oo g (^)which ^ = . . . , .The last term of the right side of Eq. (|2 

pictures a finite-size corrections to the flow field. Similarly, the diffusion factor can be drawn 

D(q,i) =- (£ 2 (1- tanh 2 (/3£- Aft* ))) ? 

+ {[/?(£(£• Aq) [1 - tanh 2 • AO*)] ) £ - q 



lim y/N 



N- 



1 N ll 2 

-^6tanhG9ei-An*)-n* | 



(21) 



Obviously, not as the system with sequential dynamics, the finite-size effects for parallel dynamics is govern by a 
homogeneous Markov process. The More intensive and detail work including verification of numerical simulation is 
ongoing. 

In short, the macroscopic dynamics of networks is all the same to both updating ways, synchronous and asyn- 
chronous, in the limit of far away of saturation. But there exist deviation in the fluctuation of pattern overlap implied 
in Eq. (|l7|). Moreover, in the condition of away from saturation, the finite-size effects in parallel dynamics is de- 
scribed by homogeneous Markov process which is not as the timc-dcpcndcnt Ornstein-Uhlenbeck process in sequential 
dynamics. 
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